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(54) A method for computational graceful degradation in an audiovisual compression system 



(57) The invention disclosed here is a method for an 
encoder to encode audiovisual information for transmis- 
sion to the decoder without any prior knowledge of the 
computational capabilities of the decoder. A descriptor 
containing parameters that can be used to estimate the 
complexity of the decoding process Is embedded in the 
system strearh. The encoder also encodes the video 
information in such a manner that the decoder can 



choose to ignore some of the information and only 
decode a subset of the encoded information In order to 
reduce the computational requirements. This method 
allows rnore than one decoder to decode the same bit- 
stream giving different resolutions depending on the 
computational capability of the decoder 
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Description 

BACKGROUND OF THE INVENTION 

1. Field of the Invention ' ■ ' 

[0001] The present invention relates to a hietHod for 
computational gracefur degradation in an audiovisual 
compression system. This invention'is- useful in a multi- ■ 
media encoding and decoding environment where the' 
computational demands for decoding a bitstream is not 
well defined. It is^aiso useful in cases where channel 
capacity Is limited and sorrie form of quality of service ' 
guarantee is required. It is also useful for inter working 
between two video services of different resolutions. 



2. Description of the Related Art 

[0002] c It is common in the case of software decoding 
to employ some form of graceful degradation when the 
system resources is not sufficient to fully decode air of* 
the video bitstream. These degradation ranges frorn' 
partial decoding of the picture elements to dropping of 
complete pictures. This is easy to implement in the caise 
of a single video stream-^ ' ' * 

[0003] In the proposed new ISO/IEC SC29/WG11 
standard of MPEG-4. it is possible to send multiple 
Audiovisual, AV. objects. Therefore, the total complexity ' 
requirements no longer depend on one single strei^m 
tHJt on multiple streams. ^ 
[0004] In compression systems such as MPEG-1, - 
MPEG-2 and MPEG-4; a high degree of temporal' 
redundancy is removed by employing motion compen- 
sation. It is intuitive to see that successive pictures in a 
video sequence will contain very similar information. 
Only regions of th^ picture that are moving will change 
from picture to picture. Furthermore, these regions usu- 
ally move as a unit with uniform motion. Motion compen- 
sation is a techi|^ue where the encoder and the 
decoder keep the pconstructed picture as a reference 
for the prediction of^e current picture being: encoded or 
decoded. The enc^er mimics the decoder by imple- 
menting a local det^der loop. Thus, keeping the recon- 
structed picture s^jiphronized between the encoder and 
decoder. W*^ 

[0005] The encoder performs a search for a block in 
the reconstructed picture that gives the closest matcKto 
the current block that is being encbded. It then com- 
putes the prediction difference between the motion 
compensated block ^ and the current block being 
encoded. Since the.motion compensated block is avail- 
able in the encodetand the decoder, the encoder only- 
needs to send the fpcation of this block and the predic- 
tion difference to t^e decoder. The location of the block 
is commonly refertid to as the motion ^vector. The pre- 
diction difference i^ jcommonly referred, to as the motion 
conrpensated pr^|^ction error.- These * information 
requires less bits tqsend that the current blogk itself." . 



[0006] in intra-picture coding, spatial redundancy may 
be removed in a sirnil&r way. the transform coefficients 
of the block can be predicted from the transforrn predic- 
tion of its neighboring blocks that have already being 
5 decoded: ■ -'-^n^' ^ ■ ' 

[0007] Thei-e are two! major problems to be solved in 
this invention. The first is how to indicate the decoding 
complexity requirements of the current AV object. In the 
case where there" are muitiple AV objects, the systems 
10 decoder must decide' how 'much resource should be 
given to' a particular object and which object should 
' ' have priority over another. In'other words, how to model 
the complexity requirements of the system. A point to be 
noted here is that the complexity requirements of the 
15 ^- decoder is dependent on the implementation of the 
decoder. '' An operation that is complex for One Imple- 
mentation may be simple for anothier ihnplerrientation. 
Therefore,- some 'foriri of irhpleme^htation independent 
complexity measure is required. 
20 [0008] The second problem is how to reduce complex- 
ity requirements in 'the decoder. This deals with the 
method bfTediicihg the complexity reqiiir^^ 
decoding process while retaining as much of the' infor- 
mation' as ^possible. One biggest problem in graceful 
• 25 degi^adation is the probienn of drift caused by errors in 
' the motion compensation. When graceful degradation is 
employed the recdnstructed - picture "is incomplete or 
noisy. These errors aris propagated from picture to' pic-' 
ture resulting in larger arxJ larger' errors. This noise 
'' 30 propagation ' is i-ef erred to' as drift' * ' ' 

SUMMARY OF THE INVENTION 

[0009] - 'In order to solve' the problems 'the following 
steps are taken in tHe present invention.'' '■ ^ 
[0010] The AV object encoder encodes the AV object 
in a manner that would allow different amounts of grace- 
ful degradation to be employed in the AV object 
decoder. Parameters' relating to the computationai com- 
plexity reiquifemerits of the AV objects are transmitted 1h 
the systems encoder. Implementation independent 
' complexity measure is achieved by sendirig parameters 
> that gives an" indication of the operations that are* 
required; • ' ' ' . - . r - r 

45 [001 1 ] At the systems decoder, estirriates* of the com^ 
plexity required are made based on these'parameteVs 
as well as the im'plemehtatidri methods being employed . 
The resource scheduler then allocates the appropriate 
amount of resources to the decoding of f he different' AV 
BO objects. In the' A^ object decoder, computational grace- 
ful degradation is employed when the resources are not 
sufficient td decode the AV object completely. ' 
[001 2] ' In accordance with a first aspect of the present 
invention, a method of encoding a plurality of audidvis- 
55 ual objects into a compressed coded representation 
suitable for computational graceful degradation at the 
decoder comprises: ? ' • " » 
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encoding said audiovisual objects,^ incorporation 
methods allowing computational graceful degrada- 
tion to be employed in the'decoder/ihto their coded ' 
representations*; , , j . . 
estimating the implementation independent compu- : 
tational complexity measures in terms of a p!ui;ality. 

of block' decoy ing pararneters; ' ' . ]\ - , 

partitibning siaid coded representations of the audi- 
ovisual objects into a plurality of. access units ^ryd 
adding header information to, form packets; 
inserting a descriptor containing said block.decpdr, 
ing paranieters into the header of the packet; and . , 
multiplexing these packets, to form a single, multi,-- 
plexed bitstream. ... _ ^ 
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[0013] In accordance with a second aspect of.. the. ( 
present invention, a rnethpd of d ecodi rig: a, multiplex^ 
bitstream,,yvith computational graceful degradati9n, ta,; 
obtained a plurality of audiovisual qbjects, cprpprj^es; , 

de-rnuitiplexrng the single.multiplex^ bitstrearn iintp^^ • . , 
a plurality pf packets comprising of packet h^ad^fs ,,-., 
and access units; ' ' ; . ,( ,. ; ijoir-- -,. . 

extracting the descriptor containing a .plurality,;. pf . 
block decoding parameters frqm the packet ihea^ii;, .^ ,: 

\ \. , , ■ - r ^i" 

reassembling the access . uoits Jntp their original^. • ; 
coded representations of th^ audipvisua). objects;. , j - . . ( 
estimating the decoder specific cprnputatiQnal*c?pni.?: ■. ^ 
plexity measures based . on said- block; decod(ng,..i:,^P; 
parameters and the current decoder irnplementa- 
tion;and ' ; . r. - » ^HA^^^iS".^ 

decoding said coded representations of the audio- 
visual objects, :uslng cornput^tional graceful, degra- . / ^ 
dation, where necessary,". to satiSify the, esji mated , os,, 
decoder specific computational, corpplexity require- : 
mentS; ^. \ • ' ' . 

[0014] . Preferably. , the incorporation method , 
computational, graceful degradation to be erpployj^d.iri v.f?P : 
the decojder, comprise; ,u . . !..:>. - , .;vr. v^rii^iK ; 

partitioning th^^^nput ptctureSjto be encpded intp^ a. 
plurality of sub-regions numbered in increasing 
order, begirining,.>^vith the full picture .^js. tjie first ?ub- 45 
region, vvh ere each subTr^gjon.ccvriprising prily pj a , 
subset pf the pixels within they sub-regionpreceding ti 
it; ' , _ . \, \ 

entropy coding position, ^ndjdfmensipn^pf, the: 
sub-regionis into a, ppmpre^ssed coded -representa- r.j50 
tion within the jbitstrea^i^i; . , j . . - . • , * . 

. further partitioning the sub-regions into a plurality of . ; 
blocks for .epcofjing .into ,a (5om|3r.essed coded rep- , 
resentatipn, within th,e bitstream:,. t > . 

performing.motioniestimation and, motion iCompen- 55 
sation for said blocks using only^the pixels from the -■•a ■ 
reconstructed picture that belong to sub^^egtons 
having the same or higher numeric order as said 



blocks; ^ » . .. 

entropy coding the motion vectors into a com- 
pressed coded representation within the bitstream; 
transforming the motion compensated prediction 
difference into an orthogonal domain; . .. 
quantizing the transformed coefficients using a 
quantization method; and, . , . : . , . 
entropy coding the quantized transformed coeffir., 
cieots . into a compressed coded representation 
..within the. bitstreani. . ... . , 

[0015] 1 Preferably, the method for.decoding the coded 
representations of the, audipvisual Qbjecls in . accord- 
ance with the seGor]d aspect., using ; computational 
graceful degradation yvhere .njecessary to satisfy , the 
estimated decoder specific computational complexity 
requirements, further comprises: . - . -t .1. ^ . .> 

entcqpy decoding the position and dimension of the 
' sub-region^ from the compressed coded represen- 
tation within the bitstream; . . 'A-r y--.^. 
selecting only the -blocKs^ that ar« withiri the sub- ■ 
region of interest for decoding; t ' • t f 1 . * - . 
entropy decoding the compressed coded represen- 
tation to give quantized transformed coefficients: 
inverse, quantizing said quaintized ..transformed . 
coefficients to give the transformed coefficients;^ * ; 
. inverse: transforming said transform ooeffibientsito 
give* the spatial domain nrK)tion compensated pre- - 
diction difference; -'ii^ v 
entropy decoding the motion vectors from the com- : 
pressed coded representation within the bitstream; 
perforHjiing > motion compensation, for. said blocks 
using pnly .the pixels from the teconstructed picture 
that belong, to sub-regions having the same^or 
highernumeric order as said^blocks;; and, r.-:. 
reconstructing the picture and ^storing said picture 
in the frame memory for prediction of the next pic- 
ture. I w- . ' , ' * . . ■ ' . 

[0016] : Prefarably, tha method in accordance with the 
first aspeqt /Of ithe ;invention, whereby incorporation 
methods allowing pomputationaJ graceful degradation to 
be employed in thie decoder, further comprises: 

partitioning the input pictures: to be encoded into a ' 
pluraUty of -sub-regions numbered, in increasing 
order, beginning with the full picture as the first sub- 
region, where each sub-region comprising only of a 
subset of the pixels within the. sub? region preceding 
it; . - > : . ■ V ■ 5 

entropy coding the positron and dimehsion of the 
sub-regions into a compressed coded* representa- 
tion within, the bitstream; ! . ' ' i 
further. partitioning^ the sub-regions'into a plurality of. 
blocks for encoding into a compressed coded rep- 
resentation within the bitstream; ' . >^. ' . • - t 
transforming sakti' blocks :tnto "an* orthbgorial 
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domain; ' ' - • . - 

quaritijzihg the transformed . coefficients using a 
quantization method;' 

performing quantized transform cbefficient predic- 
tion for said blocks using only the corVespbnding 
quantized transform coefficients from the blocks 
above arid to the left that belong to "yub- regions 
having'the same or Hiigher numeric order as said 
blocks; and, ■ -i » -r. , - , . . 

enfropy coding the predicted diff erehce of the quan- 
tized transformed coefficients into a compressed 
coded representation within the bitstream ' 

^ ' *■ - ■ t ■ ■ . . 1 , 

[0017] - Preferably, the' method in accordance with the ' 
first aspect'of the invention, comprises:^ ■ ' » 

- ' * * ■ i . i' -f • 
entropy decoding the position arid dimension of the^ ^ 
sub-regions from the compressed coded represen-'>' 
tatron.within the bitstream; ■ . • - i . >. 
selecting- only the blocks that are within the sub- 
region of interest for decoding; ^ • .V . ■..> 
entropy decoding the compressed coded represen- 
tation' Ip give quantized transformed- coefficients; ^ 
performing quantized transform coefficient predic- 
tion for said blocks, using only the corresponding 
quantized transform coefficients from the blocks^ 
above- and- to . the left that belong to sub-regions 
having the sameior higher' numeric order' as said' 
blocks;* ' . - , . ; , . 

inverse, quantizing 'said quantized transformed 
coefficients to give the transformed coefficients;' ; 
inverse: transforming said transform .coeff icients to^= 
give the spatial domain pixel values; and; ' ^ 
reconstructing the picture and storing said picture 
in the frame memory for^prediction of the next pic- 
ture. . r - . - 

[0018] Typically, the plurality of block decoding param- 
eters comprises numeric numbers indicating -the 
number of: . ' . „ , , , , , ^ , 
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35 



block ent'opy decoding operations; / 
block motion compensation operation; ; 
block inverse quantization operations; ' 
block transform operations; . \ ^ ; 

block addition operations; and, • • ■ - . 
block memory access operations. < . . 

[001 9] - Preferably, the descriptor comprises:' • > 

a descriptor identification number* signaling the 
descriptor typ|; \ > ii. 
a descriptor length field to indicate the size of the 
descriptor; anil, / t . . - - 

a plurality of bjock decoding parameters. ' - 

[00201 • Typically, In the method of partitioning the input 
pictures to be encoded into a plurality of sjjb-regions, 
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50 
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the sub-regions are rectangular. '''' * ' ' ^ 

[0021 ] Preferably, iri the method of performing motion 
. estimation and motion coriipensation for said blocks, 
, using only the pixels from the reconstructed picture that 
.belong to sub-regions having the same or higher 
numeric 'order as Wid'%!oc^^^^^^ implies that only predic-' 
tron blocks tfiat lie corhpletely' within said' sub-regions 
are sele6t'ed. ' ' ^ • ' ^ ' 

[0022] ■ Typically, when only the pixels from the recori- 
. strijcted pictiljre that belong to sub-regions' having the 
same or Higher numeric Arder as said blocks are useb, 
predictiori'^bldcks may lie partially butside said 'sub'- 
regions but vJrith the additional condition that'the pixels ^ 
lying outslde'said sub-region are replaced by the near- 
est pixels^ from within the sub-regions. ^ '* ' * 

■ [0023] - Preferably, in the method of partitioning the pic- 
tures into' a plurality of sub-regions, the pdsition and 
dimension 'of each of said sub-regions may vary from 
picture to picture' and said position and said dimension 
are coded^ by mean's^ of a pari scan vector,' gi'vihg the 
horizontal and vertical displacement, a width and a 
heights ' - ■ ^ ^ . . - . ; , ; 

[0024] Typically, in the method of partitioning the pic- 
tures into a plurality of sub-regions, the 'position and 
dimension^ of the sub regions are the same fi-bm picture 
to picture and said position*^ and said dirnension are 

■ coded once at the beginning of the sequence by means 
' of a horizontal and vertical displacerhent. a width arid a 
' height. ' *^ * ^ ; • 

[0025]^^^'^ Pr'efer'ably,' in the method of encoding and ' 
- decoding, the transform is the Discrete Cosine Trans-- 

form, -^i^' ■ ■ . ; ; ; - 

[0026]' ^ Typicaliy. in the method of encoding and decod- 
' ing, the number of siib-regibns is two. ' ' • ^ • 
[0027]! ■ Preferably, in the' method where there is a plu- 
rality of "sub-region numbered in increasing order and 
the motion vector can point into a sub-region of lower 
order but not out of a lower order to a higher ordered = 
' number;* : '-'j - . ' .■ - i 

BRIEF DESCFtrPTIONOFTHE DRAWINGS ■ ^ 

[0028] 

Figure f is an overall block diagram of the prd^ent 
invention; 

Figure 2' shov/s a block diagram of encoder and 
decoder of the* present invention;' 
Figure 3 illustrates the embodimerit of the sub- 
region and the ^ motion v/ector restriction of ^ the 
present invention; - . - ^ . 

Figure 4 illustrates the embodirrient for the pan- 
scan vectors and the sub-region dimensions in the 
present invention; - ' . . . — 

Figure 5 illustrates the second 'embodiment for the' 
padding method of the motion compensated predic- 
tion at the sub-region bbundary ; and, t • ^ . m . 
Figure s iliUstrafes the block diagrarn for^the Com- 
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plexity Estimator. , ^ ^ . ^ , . 

descriptVon of the preferred embddi- ' ■ . ' 
MENTs ^ . . . \.!; . i^V 

[0029] Figure 1 shows an pvera I i . system . 
gram of the present iny^ntion.' ^Encoder unit 110.». 
encodes the video sequence to allow corhputationaL 
graceful degradation techniques. The output of encoder . . , 
110 is a coded representation of the. video sequence'. ; to 
that is applied to an encoding buffer 120. At tt}e same 
time the. video, sequence and the coded representatioOr . 
are also applied to a complexity parameter encoder: .1^ , 
where the parameters associated with the operatlori 
that are required for decoding is computed and i5 
encoded. .These. information together wjth the output! of .- - • 
. the encoding -.buffer. 1 20. are. passed to / a ,,,Systern. , ^ . 
Encoder and Multiplexer unit 140, where a systemrmuiti-., , . . 
plexed , strearn Is , formed, The: system-rnultiplexed.- j . 
stream js transmitted through a. transmission media 20 

^50. _ , \ [ .. . -V^-i-.T^.^ ..L..,-. 

[0030] A Demultiplexer and System Decoder unit 160 j, ;: , 
receives the systemj-rnultiplexedi strearn-, vyherje the ;bit-; > •* - 
stream Is, demultiplexed into its respective el erneaitary 
streams. The video el ementar.y stream is passad to .a . ^ 
Decoding Buffer 170, and complexity pararneters are- 
passed, to, a Scheduler and Complexity-Estimator:, unit - 
180. From the Decoding Buffer 170, the video elemen- - 
tary stream is passed to a Decoder unit 190. The; \ 
decoder 190^ waits, for the cornmarids coming frprn thef.; 
Scheduler unit 1 80 before decoding.. .... , ; . r ; ^tyi , 

[0031] The Complexity Estimator 180 gives the 
amount of decoder, computational graceful .degradation; -j 
that is to be employed. CornputationaKgraceful:C!egrar,, i 
dation is achieyed in the decocier by. decoding only a 
sub-region of ,tha .cpmplete picture that is deemed I0 
contain the more .important information. The encoder 
will have to preventihe encoder and decoden fronn drift- 
ing apart under these conditions. After decoding, the..i. 
decoder unit 190 also feedback information to the 
Scheduler and Complexity Estirnator 18Q sq ttiat the ; 
information may be used to estimate the complexity of 
the next picture. u: 
[0032] The following is the embodiment of the various 
units illustrated in the above invention shown in Figure , 
1. . ' t. .r. • 

[0033], f^igufe. 2 is a block diagram of the enqoder and 
decoder according to the present , embodirnent. The 
input picture to the encoder 1 10 .ds. segmented into 
blocks for processing. Temporal redundancy is removed . so 
from the picture by subtracting the motion cornpehsated 
picture of the previous picture, from the^current picture.; 
The prediction difference is then ^transformed into the 
DCT domain in a OCT unit 111. The resulting DCJ coef: 
ficients are then quantized in a .Quantization unit 112..( 55 
The quantized coefficients are then entropy coded in as 
Variable Length Coding (VLC) un:t.1 13 to form:the com- 
pressed output bitstrearri- Tha encoder- -1 l O^Jso has a 
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local decoder loop comprising of an Inverse Quantiza- 
tion unit 1 14, ah Inverse, DCT unit 1 15, a Frame Storage 
116. arid a Motion Compensation unit 117, The local . 
decoder bops mirnics the. decoder operjatipns .by 
inverse quahtizihg the coefficients and transforming it 
back into the spatial dorhaih in the Inverse Quantization 
unit 114 and Inverse DCt. unit. 115. The output is then 
added to the. output of the M6t(on . Compensated unit 
117 to form 'the reconstructed picture, this picture is 
stored iri, the Frame Storage 116 for motion compense- 
tion of the next picture. ^ . , . . 

[0034] In this embodiment .the encoder units of. Motion 
Estimation unit 1 1 8 and Motion Compensation unit 1 1 7 
are changed so that computational graceful degradation 
may be performed in conjunction with, the motion com- 
pensation without causing drift. 

[0035] i 1 Figure 3 illustrates the present invention, 
according Jo..which the picture is divided into two parts ^ 
220 and 210. the first part 220 is a subrregion that must 
be decoded jn.the decoder, regardless of whether com- 
putational graceful degradation is employed.or not. The . 
second pa/t 210 is the region outside of the. sub-region, 
which may be discarded by the decoder wherv computa- 
tional graceful degradation is employed.. ^ 1 &u<-- ? -.^ 
[0036] , Figure 3 also show two blocks that are used for 
motion eocnpensalion. When motion compensatipri is 
performed on a block 250.that resides in the subrregion. 
220. the. motion compensated prediction bloek^ must- 
also come from within the sub-region 220 of the refer- .' 
ence picture. In other', wprds the : motion vector 260 
pointing out of.lhe regioais not allowed. This is referred 
to restricted motion vector. On the other haind/when a 
block 230 resides outside the sub-region 220. the 
motion, compensated prediction block can come from 
anywhere in the reference picture. This is the same as 
where there is no sub-region. \- - . 

[0037] Figure 4 shows a method how to indicate the 
sub-region 220 within, each. picture. In order to specify, 
the rectangular -sub-region 220 for each picture the fol- 
lowing parameters must be specified for each picture 
and be encoded in the picture header of the compress 
bitstream. In Figure 4, a picture 310 and the sub-reglbn ■ 
220 is illustrated. The horizontal offset 330 of the left 
edge of sub-region 220 from the left edge of the picture; 
and the vertical offset 340 of the top edge of the sub- 
region 220 from the top edge of the picture are shown- 
These two parameters, referred to as the pan scan vec- 
tors, are used to indicate the location of the sub-region. 
The width '350 and the. height 360 of the sub-region 220 
are the second set of parameters that are required to 
specify the dimensions of the sub-region 220. 
[0038] In a second embodiment of this invention, the 
motion .vector for a block, rn, the sub-region need not be 
restricted. It is allowed to point out of the sub-region of 
the reference picture. However padding is required. This 
is illustrated in Figure 5 in which the picture 310 and the 
sub-regicn 220 are shown. The motion compensated 
prediction 430 is shown straddling the boundary of- the 
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sub-region 220. A portion 431 of the block residing out- 
side of the sub^region 220 is not used for prediction and 
is padded by repeating the value of the pixel found at 
the edge of. Ihe sub-region 220. A portion 432 of the 
block residing in the sub-region 220 is used without any 5 
padding. A similar padding method is used for the^rows 
and columns fpr blocks located at the vertical edge and 
horizontal edge, respectively. ■ v. r ~ 

[0039] Like the first embodiment; the method accord- 
ing to the second embodiment Would also enable com- 10 
putational graceful degradation method to discard the 
portion of the picture outside the sub-region 220 without 
causing the encoder and decoder to drift apart. ' ' 
[0040] ' i Apart -from motion compensation that may 
cause drift in inter blocks, intra blocks at the top and left 15 
boundary of the sub-region 220 are also restricted from 
using any blocks 'outside of the sub-region 220 for pre^ ' 
diction. This is because in the computational s graceful 
degraded, decoder, these blocks would not be decoded" 
and thus ,the prediction cannot be dupliqated. 'This pre- 20 
eludes the commonly used DG and AQ 'cpefficient pre- 
diction from being employed in the encoder. 
[0041] .Figure 2 also illustrates a block diagram of a 
decoder 190: . The ^embodiment of the decoder - 190. 
employing ; computational graceful degradation is 26 
described, here. The compressed bitstream is received 
from the transrnission and is passed to a Variable 
Length Decoder unit 19t ' where the bitstream is 
decoded according .to. the syntax and entropy method 
used. The decoded. information is then; passed to the 30 
Computational Graceful - Degradation Selector ^92 
where. the decoded information belonging to tfie sub- 
region 220 is retained and the decoded informatidn out- 
side of the sub-region . 220 is discarded. The retained 
information is- then passedrto an Inverse Quantization 35 
unit 193 where the DCT coefficients are recovered The 
recovered coefficients are then passed to an tnverse 
DCT unit ,194 where the coefficients are. transformed 
back to the spatial domain. The motion compensated 
prediction is then^ added to form tfie reconstructed pic- 40 
ture. The reconstructed, picture Is stored in? a; Frame 
Storage 195 where it is used for the prediction of the 
next picture. A Motion compensation unit 196^ performs 
the motion , compensation according, to the same 
method employed in the encoder l ljO. . : i 45 
[0042] In the first embodiment of the encoder where 
the motion vector is restricted no additional modif ication 
is required in the decoder. In the second embodiment of 
the encoder where the motion vector is not restricted, 
the motion compensation method with padding so 
described above in connection with Fig. 5 is used in the 
decoder. Finally, intra blocks at the top and left bound- 
ary of the sub-region. 220 are also restricted from using 
any blocks outside of the. sub-region 200 for prediction. 
This precludes thei commonly used DC and AG coeffi- ss 
cienl prediction from being employed. . - . 

[0043] In this emlDodiment the Complexity Parameter 
Encoder consist, pf. a counting unit that counts the 



number of block decoding 'operations that are required. 
The bfock decoding operations are not basic arithmetic 
operations* but rathef a collection of operations that are 
performed on a block.' A' block decoding operation bah 
be a block inverse quantization operation, a blobk 
inverse DGT operation, ablock me'hriory access or some 
other collection of operations' that perform some decod- 
ing task oh the block by 'block basis. T he Cohnplexity 
Parameter Encoder counts the number of blocks that 
require' each set of operations and indicate these in the 
parameters. The reason* block decoding ojDerationis are 
used instead of simple arithmetic operations :s because 
different'implementations may implement different oper- 
ations more efficiently than others. ' ' 
[0044] " There is also a difference in decoder architec- 
ture and different amounts of hardware and software 
solutions that makes the' Uise of raw processing power' 
and memory acdess measures unreliable to indicate the 
complexity requirements. However, if the operations 
required are indicated by^parameters that'counts the 
numben-of^ block 'decoding' operations '^necessary, the 
decoder^ can estimate the complexity. This is because ■ 
the decoder knows the amouht of operations required^ 
for each of : the block decoding operations In its-' own 
implementation: : ' . - ,f - , - 
[0045] In the embodiment of the System Encoder and 
Multiplexer, the elementary bitstream are packetized 
and multiplexed for transmission. The information asso- 
ciated with the complexity parameters is also multi- 
plexed into the bitstream. This information is inserted 
into the header of the packets. Decoders that do not 
require such information may simply skip over this' infor- 
mation. Decoders that require -such- information -can 
decode this information and interpret them to estimate 
the complexity requirehients. ' = 1, - 
[0046] In this- embodiment the encoder inserts the 
information in the form of a descriptor in th^^ header of 
the packet. The descriptor contains an ID to indicate the 
type of descriptor it is followed by the total number of 
bytes .'Contained : in the descriptor. The rest of the 
descriptor contains the: parameter for each of thferblock 
decoding operations. Optionally the descriptor rhay also 
carry;, some user, defined- parameters that are not 
defined earlier. ^ - t ^ . i -i. . - -\- 
[0047] In the Scheduler and Complexity Estimator v180 
in Figure 1/ the time it takes for decoding all the audio- 
visual objects is computed based on the parameters 
found in the descriptor as well as the feedback informa- 
tion from the decoder.* s ' .V , - fs. 
[0048] An embodiment of the Complexity Estimator 
180 is shown in Figure 6. The block decoding operation 
parameters: 181a. 181b and 181c are passed into the 
complexity estimator»483.after'being pre^ multiplied with 
weightings 182a, '182b and ;182c. respectively. .The- 
complexity estimator 183 then estimates the complexity 
of the picture to be decoder and passes the' estimated 
complexity 1B4 to the decoder 190. After decoding the 
picture the decoder *,190 returns the actual complexity 
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1 85 of the picture! An error 1 86 in the jCornplexlty esti- . . , . 
mation. Is obtained , by taking a difl^Qrenee between tKe, . 
estimated cqmplexiiy 1 84 and the.actual complexity 1 85 
of the picture. , The , error 186..is, then passed into the . . 
feedback gain unit 187 wh^r^ jthe. corrections 188av 5* 
188b and 188c to the weightings.,are fqur;id; The weights . , 
are then modified by these corrections and the process . , 
of estlmatirig.the complexity of th^e next picture cpntin- . 
ues. ■ , ; • , , , . , . . ; 

[0049] The effect of this inv,enti6n is that the need for. ; io 
Implementations that can handle the worst case is^no ~ 
longer necessary. Using the indications of computa- . , 
tional complexities and the computational graceful deg-r. i 
radatlon methods simpler decoders, ican: be n/, 
implemented. The decoder ,would haverthe-capabjlitie? <..:.7,5 
to decode most pf the sequences, but if it enqpunters ^ , 
some more derrianding sequences,; if .cari. degrad.e the , . - 3 
quality and ^resolution of .the decoder, output In, order-to- 1- - 
decode the bitstr earn. • . < , .'.v/? ..u ,::- > 
[0050] .1 Thls.invention Is also useful for inter working of : 20 
services that have different . resolutions* and/or different * • ■ - 
formats: The sub-region can be decoder.by the decoder 
of lower resolutions where asthe decoder of big her. res- : 
olutions can decode the full picture. One example is; the- r 
inter working between 16:9 and 4:3 aspect ratio decod- 25 
ers- : ... ■ ;r <• A i ■ 



Claims 




1. A. method for encoding a plurality of. audiovisual 30 
objects into a compressed, coded representation o. 
suitabla.for computational graceful .degradation atf, ^ 
the decoder, comprising: 1...^ * / * *; 

encoding said audiovisual objects, incorpora- 35 
tiori methods allowing computational graceful , j 

, .degradation to be employed in the decoder. .« 

. into their coded representations; . - • - ' 
.estimating the- implementation independents 
computational complexity measures in terms of >4o 
a plurality of block decoding parameters; h < m , ^ 

. : partitioning said. ceded, representations of the 
audiovisual objects into a^ plurality of access, 
units and adding header information to form ^ 

V packets;- —1 t , : ; .;. . . - j (\ > 45 
' inserting a descriptor containing said block'. 

. deeoding-parameters into the header of the 
> packet; and - - .... .. j -;^,. • • / • 

multiplexing these packets to form a single mul-^ ^ • 

: tiplexed bitstream. . . so 

. , -v. •( ^ ■ . .> ■ • j 4. 

2. A. method of decoding a multiplexed bitstream, with f i 
computational. graceful degradation, to obtainedia. 
plurality of audiovisual objects, comprising: * 

^ *; , _ - i • ' ' . ■ . • yi: i; • ; . t" ..rSS 

. defmultiplexing .- the single multiplexed ' -bit- 
.> , istreamjnto a' plurality of packets comprising of K*: 
. -^ packet headers and access units; ■ ^ • 



(. extracting the descriptor. containing a plurality . 
of block decoding parameters from the . packet 
headers; , . , , . , .' « . --^ 

reassembling the access units into their origi- - 
, nal coded representations of the audiovisual 
■ objects; - - . ' * . <* • • 

. . estimating the decoder specific computational ' ''' 
complexity, measures based on said * block ' 
. decoding parameters and the current decoder : . 
( . implementation; and ■ . ■ . ^ .-j - 

. . jdecodirig said coded representations . of the 

• audiovisual objects, .using computational ■ 
graceful degradation.^where necessary, to sat- 

. ,isfy the estimated decoder specific computa- * 
tional complexity requirements. .1 . • 

A method of encoding said audiovisual objects into 
their compressed coded representations, according « 
to claim 1 ; whereby incorporation methods allowing - 
computational graceful degradation to be employed ' ' 
in the decoder, further comprises: ^ > 

partitioning the input pictures to be encoded ^ - 

into a plurality-'Of sub-regions numbered in 
increasing order, beginning with the full picture . l^vm 
r,. as the first sub-region, where each sub-region- ' - ^'•.s?^ 
: comprising only of a subset of the- pixels within -^^-^ 
^.i 'the sub-region preceding it: ' * ' ^^^^ 

I ..entropy coding the position and dimension of ' ^.-i^k 
V. /.i the sub-regions into 3: compressed coded rep- < 
s * • resentation within the bitstream- ' ^ r - ; 

further partitioning the sub-regions into a plu- y 
i.raiity of 'blocks for encoding into a compressed .-vv-.^^c^ 
. coded representation within the bitstream; 'sm^T 
performing motion estimation and motion com- ; t^^;^^- 

• pensation for said blocks using only the pixels " .^f? -r^ * 

from the reconstructed" picture that belong to • 
■sub-regions having the same or higher numeric ' ^ 
. u order as said blocks; - 

entropy coding the motion ve(:;tors into a com- 
:i . pressed boded representation within the bit- * 
- ''Stream;- ^ J; • ' - ■ ■* 

'transforming the motion compensated predic- 
f i tion difference into an orthogonal domain; 

quantizing the transformed coefficients using a' - ' 
. quantization method; and 

entropy coding the quantized transformed coef- ' ' 
ficients into a corrpressed coded representa- 

* tion within the bitstream.' * . » - • • - 

A hiethod for decoding said coded representations 
of the audi o\>isual'f objects according td;* claim 2, ' 
using computational* grBceful degradation, where' 
necessary.ito satisfy the estimated decoder specific' 
computational complexity -requirements ' further 
comprising: • • • ' ' ' ■• ' ' 

' entropy decoding the position arid dimension of * ' 
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the sub-regions from the compressed coded 
representation within the bitstream; 
selecting only the blocks that are within the 
sub-region of interest for decoding; 
entropy decoding the compressed coded rep- s 
resentation to give quantized transformed coef- 
ficients; 

inverse quantizing said quantized transformed 
coefficients to give the transformed coeffi- 
cients; 10 
inverse transforming said transform coeffi- 
cients to give the spatial domain motion com- 
pensated prediction difference; 
entropy decoding the motion vectors from the 
compressed coded representation within the is 
bitstream; 

performing motion compensation for said 
blocks using only the pixels from the recon- 
structed picture that belong to sub-regions hav- 
ing the same or higher numeric order as said 20 
blocks; and 

reconstructing the picture and storing said pic- 
ture in the frame memory for prediction of the 
next picture. 

25 

A method of encoding said audiovisual objects into 
their coded representations, according to claim 1 . 
whereby incorporation methods allowing computa- 
tional graceful degradation to be employed in the 
decoder, further comprises: 30 

partitioning the input pictures to be encoded 
into a plurality of sub-regions numbered in 
increasing order, beginning with the full picture 
as the first sub-region, where each sub-region 35 
comprising only of a subset of the pixels within 
the sub-region preceding it; 
, entropy coding the position and dimension of 
the sub- regions into a compressed coded rep- 
resentation within the bitstream; 40 
further partitioning the sub-regions into a plu- 
rality of blocks for encoding into a compressed 
coded representation within the bitstream; 
transforming said blocks into an orthogonal 
domain; 45 
quantizing the transformed coefficients using a 
quantization method; 

performing quantized transform coefficient pre- 
diction for said blocks using only the corre- 
sponding quantized transform coefficients from 50 
the blocks above and to the left that belong to 
sub-regions having the same or higher numeric 
order as said blocks; and 
entropy coding the predicted difference of the 
quantized transformed coefficients into a com- ss 
pressed coded representation within the bit- 
stream. 



6. A niethod for (Jecbdihg said codied representations 
of the audidvisuar objects according to claim' 2, 
using computational graceful degradation,' wher-e 
necessary, to satisfy the estimated decoder specific 
cohnputatiohal complexity* requirements further 
comprisihg: ' ' ' ' « -'wmu • 

entropy decoding the position and diniension of 
the sUb-regions' from the cbmpressed coded 
repfresentation within the bitstream; 
selecting only the blocks thalt are within the 
•sub-region of interest for decoding; '' ' ' '* ' 
' -entropy decoding the compressed coded rep- 
resentation to give quantized transformed coef- 
ficients;'' ' J ^ ' ' • ' ■ . . 

^perforhiing quantized transfoVm" coefficient pre-' 
diction^ for said' blocks using only the corre- 
sponding quantized transform coefficients from 
the blocks above and to the left that belong to 
sub regions having the same or higher humenc 
order' as said blocks; - ^ 

inverse quantizing said quantized transforrhed 
coefficients to give the transforhied coeffi- 
cients; ^ ^ ' r ' K i^f - : . . 
inverse transforming said transform coeffi- 
cients to give the spatiai domain pixel* values; 
and 

reconstructing the picture- and storing said pic- 
ture in the frame memory for prediction- of the' 
nextpicture. . , ; ■ ^ - :r 

7.. A methods for ^estimating the implementation Ihde- ' 
pendent computationial complexity measures; as in 
claim 1, whereby' the plurality of block decoding 
parameters comprises numeric numbers indicating 
the number of • --^ i . - > 

block ientropy decoding operations; ' - • 
block motion compensation operation; 
block inverse quantization operations; >^ ^ ' 
block transform operations; ' 
block addition operations; and 
block memory access operations. 



8. A method of encoding the block decoding parame- 
ters in the header of the packet as in clalm 1 , where 
the descriptor compris;es: i . " ■ ' ' ■ 

a descriptor identification number signaling the 
descriptor type; 

a descriptor length field to indicate the size of 
the descriptor; and 

a plurality of block decoding parameters. 

9. A method of partitioning the Input pictures to be 
encoded into a plurality of sub-regions according to 
claims 3 and 5, where said sub-regions are rectan- 
gular. 
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10. A method of .perforn^ing ,moto , . . 
motion compensation fpr said blocl^s according, to 
claim 3, whereby using; prijy the pixels from the;;.ri, 
reconstructed picture th^t belpng/ito sub-regions 
having the same or higher ^rjujmeric order as said/.^ 5 
blocks, implies that only prediction blocks .that^ lie,^ ^ 
completely within said sub-regions are selected. 

11. A method of performing mot(on estimation and 
motion compensation for said^ blocks according to 10 
claim, 3, whereby using only the pixels from the 
reconstructed pictur^ that belong to sub- regions 
having ttie same or higher, numeric order as said 
blocks, irnplies thaj prediction blocks may Ijp .par- 

. tially outside said sub-regions but with the -addi- is 
tional condition that the pixels flying* outside said 
sub-region are replaced by the nearest pixels from 
within tl;ie. sub-regions, . .,.M*i; ' ., * 

1 2. A method of partitioning thepictures intOia plurality 20 
of sub-regions according to claims 3 ancji. 5, where 
the ppsitipn and, dimension of each, of said subr 
regions may vary from 1 picture to picture and isaid 
position and said dimension are coded by meams of 

a pan scan vector, giving the horizontal and vertical 2s 
displacement., a width ^and a height J . 

13. A method.of partitionlng- the pictures.into a plurality 
of sub-regions according to claims 3 and 5, 'where 
the position and dimension of the sub region^ are 30 
the same from picture to picture and said position 
and said dimension ar^e coded once at the' begins n ^\ 
ning of the sequence-by means of a ihorizontat andvix- ' 
vertical displacement,, a wLdth^and a height. ' • f n> 

14. A method of encoding and decoding according to ' 
claims 3, 4, 5 and 6, where the transform is the Dis- 
crete Cosine Transform. ^ , . ^ . * ■ 

15. A method of encoding; and decoding according -to 40 
claims 3, 4, 5 and 6, where the number of sub^ 
regions is two. - • i'-r^ 

^i,^ r-'- ' ^ • ^i..^ 

16. A method where there is plurality of sub-region 
numberedin increasing order and the motion vector 45 - 
can point into a. sub region of lower border but not out • 

of a lower order to a higher ordered number. • 
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(57) The invention disclosed here is a method for an 
encoder to encode audiovisual information for transmis- 
sion to the decoder without any prior knowledge of the 
computational capabilities of the decoder A descriptor 
containing parameters that can be used to estimate the 
complexity of the decoding process is embedded in the 
system stream. The encoder also encodes the video 
information in such a manner that the decoder can 

Fig.l 



choose to ignore some of the information and only 
decode a subset of the encoded information In order to 
reduce the computational requirements. This method 
allows more than one decoder to decode the same bit- 
stream giving different resolutions depending on the 
computational capability of the decoder. 
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